Occupant observation device

ABSTRACT

An occupant observation device includes an imager configured to capture an image of a head of an occupant of a vehicle; an eye detector configured to detect at least a part of eyes of the occupant in an image captured by the imager; a mouth detector configured to detect at least a part of the mouth of the occupant in the image captured by the imager; and a condition estimator configured to estimate a condition of the occupant on the basis of a detection result of the eye detector and a detection result of the mouth detector, in which the condition estimator changes a ratio of reflecting each of the detection result of the eye detector and the detection result of the mouth detector in an estimation of the condition of the occupant, on the basis of the detection result of the eye detector or a result of a process performed on the basis of the detection result of the eye detector.

CROSS-REFERENCE TO RELATED APPLICATION

Priority is claimed on Japanese Patent Application No. 2019-047661,filed Mar. 14, 2019, the content of which is incorporated herein byreference.

BACKGROUND Field of the Invention

The present invention relates to an occupant observation device.

Description of Related Art

Research is being conducted on a mechanism which detects a condition ofan occupant, including a driver of a vehicle, using a device. Thecondition of the occupant includes whether the occupant feels sleepy, adegree of concentration, an emotion, and the like. Important factors fordetecting the condition of the occupant are conditions of the eyes.Therefore, devices that capture an image of an occupant with a camera,analyze the image, and observe the condition of the eyes have been putinto practical use.

For example, Japanese Patent No. 5343631 discloses a driving supportdevice which is equipped with an imager that captures an image of adriver's face; a detector that detects movement of a driver's mouth orhand and opening and closing of the eyes from a face image captured bythe imager; a first feature detector that detects a yawn from a changein a shape of a vertical component of the mouth detected by thedetector, detects a sigh or a deep breath from a change in the shape ofa horizontal component of the mouth, detects a movement motion of adriver's neck or head from a change in shape of the mouth, and detects amotion of a hand that approaches or separates from the driver's face; asecond feature detector that detects an eye closing rate from a closingtime of the eyes; and a determining unit that determines an arousalcondition of the driver, a struggle or conflict condition withdrowsiness, an initial condition of dozing, and a dozing condition fromtemporal changes in the feature motions detected by the first featuredetector and the second feature detector.

SUMMARY

However, in the related art, since a proportion by which the conditionof the eyes and the movement of the mouth are reflected in estimation ofa condition of the occupant is uniform, in some cases, an accuracy ofthe condition estimation may be reduced.

The present invention has been made in view of such circumstances, andan object thereof is to provide an occupant observation device capableof maintaining a high accuracy of condition estimation of an occupant.

The occupant observation device according to the present inventionadopts the following configuration.

(1): An occupant observation device according to an aspect of thepresent invention is equipped with an imager configured to capture animage of a head of an occupant of a vehicle; an eye detector configuredto detect at least a part of eyes of the occupant in an image capturedby the imager; a mouth detector configured to detect at least a part ofthe mouth of the occupant in the image captured by the imager; and acondition estimator configured to estimate a condition of the occupanton the basis of a detection result of the eye detector and a result ofthe mouth detector, in which the condition estimator changes aproportion by which each of the detection result of the eye detector andthe result of the mouth detector are reflected in an estimation of thecondition of the occupant, on the basis of the detection result of theeye detector or a result of a process performed on the basis of thedetection result of the eye detector.

(2): In the aspect of the aforementioned (1), the eye detector maydetect at least a part of a contour of the eyes of the occupant, theoccupant observation device may further include an eye opening ratederiving unit configured to derive an eye opening rate of the eyes ofthe occupant, on the basis of a positional relationship of a pluralityof feature points in the contour detected by the eye detector, thecondition estimator may estimate the condition of the occupant on thebasis of the eye opening rate of the eyes of the occupants and theresult of the mouth detector, and the result of the process performed onthe basis of the detection result of the eye detector may be a degree ofeye opening of the eyes of the occupant obtained by the process of theeye opening rate deriving unit.

(3): In the aforementioned aspect (2), when a condition in which thedegree of eye opening of the eyes of the occupant is equal to or lessthan a predetermined degree continues for a predetermined time or more,the condition estimator may change the ratio of reflecting the detectionresult of the eye detector and the result of the mouth detector in anestimation of the condition of the occupant.

(4): In the aspect of aforementioned (1), when the ratio of reflectingthe detection result of the eye detector and the result of the mouthdetector in an estimation of the condition of the occupant is changed,the condition estimator may reduce the ratio of reflecting the result ofthe eye detector in an estimation of the condition of the occupant.

According to the aforementioned aspects (1) to (4), it is possible tomaintain a high accuracy of estimation of a condition of the occupant.

According to the aforementioned aspect (3), the accuracy of the occupantcondition estimation can be maintained high even for an occupant withsmall eyes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram showing an example of a configuration and a useenvironment of an occupant observation device.

FIG. 2 is a diagram showing a position at which an imager is installed.

FIG. 3 is a diagram schematically showing the details of processesperformed by an eye detector.

FIG. 4 is a diagram for explaining a process of an eye opening ratederiving unit.

FIG. 5 is a diagram showing a part of a person in which a predeterminedevent is likely to occur.

FIG. 6 is a diagram showing a part of a person in which a predeterminedevent is likely to occur.

FIG. 7 is a diagram showing a part of a person in which a predeterminedevent is likely to occur.

FIG. 8 is a flowchart showing an example of a flow of a processperformed by the image processing device.

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment of an occupant observation device of thepresent invention will be described with reference to the drawings. Theoccupant observation device is mounted on a vehicle. The vehicle is afour-wheeled vehicle, a two-wheeled vehicle, or the like. In thefollowing description, it is assumed that the vehicle is a four-wheelvehicle. Although the vehicle is assumed to be a right-hand drivevehicle, when applied to a left-hand drive vehicle, a left-rightrelationship in the following description may be read in reverse.

FIG. 1 is a diagram showing an example of a configuration and a useenvironment of the occupant observation device 1. The occupantobservation device 1 is equipped with, for example, an imager 10 and animage processing device 20. The image processing device 20 is equippedwith, for example, an eye detector 24, a mouth detector 26, an eyeopening rate deriving unit 28, and a condition estimator 30. Theoccupant observation device 1 estimates, for example, the condition ofthe occupant of the vehicle, and outputs the estimation result tovarious in-vehicle devices 100. The occupant may include at least adriver, and may include an occupant of a passenger seat. The variousin-vehicle devices 100 are a driving support device, an automaticdriving control device, an agent device, and other devices, and theoccupant observation device 1 estimates and outputs the condition of theoccupant according to the types and purpose of the various in-vehicledevices 100. The condition of the occupant includes some or all ofdrowsiness, a direction of a visual line, an emotion, and the like. Inthe following description, it is assumed that the occupant observationdevice 1 estimates the drowsiness of the occupant.

FIG. 2 is a diagram showing a position at which the imager 10 isinstalled. The imager 10 is installed, for example, in a central part ofan instrument panel of the vehicle, and captures an image of at leastthe head of the occupant of the vehicle. In the drawing, SW is asteering wheel, DS is a driver's seat, and AS is a passenger seat.

The imager 10 includes, for example, one or both of an infrared camerathat captures an image of infrared light and an RGB camera that capturesan image of visible light as a color image. More preferably, the imager10 includes at least an infrared camera, and may further include an RGBcamera. In the following description, it is assumed that the imager 10exclusively includes the infrared camera, and a case of including theRGB camera will be described later.

Returning to FIG. 1, each unit of the image processing device 20 will bedescribed. The constituent elements of the image processing device 20are realized, for example, by a hardware processor such as a centralprocessing unit (CPU) that executes a program (software). Some or all ofthese constituent elements may be realized by hardware (a circuit unit;including circuitry) such as a large scale integration (LSI), anapplication specific integrated circuit (ASIC), a field-programmablegate array (FPGA), and a graphics processing unit (GPU), or may berealized by cooperation of software and hardware. The program may bestored in advance in a storage device such as a hard disk drive (HDD) ora flash memory (a storage device having a non-transitory storage medium)or may be stored in a removable storage medium such as a DVD or a CD-ROM(a non-transitory storage medium), and may be installed by attaching thestorage medium to a drive device.

With respect to the function of the eye detector 24, a method ofdetecting eyes after extracting an edge will be described as an example.The eye detector 24 extracts, for example, the edge in an image capturedby the imager 10 (hereinafter, referred to as a captured image). Theedge is a pixel (or a pixel group) in which a difference in pixel valuebetween the pixel and its surrounding pixels is larger than a reference,that is, a characteristic pixel. The eye detector 24 extracts the edgeusing, for example, an edge extraction filter such as a SOBEL filter.The use of the SOBEL filter is merely an example, and the eye detector24 may extract the edge on the basis of another filter or an algorithm.

The eye detector 24 detects at least a part of the eyes of the occupantin the captured image, for example, on the basis of the distribution ofedges extracted in the captured image. The eye detector 24 may directlydetect a part of the eyes (or a feature point to be described later), byinputting a captured image to a learned model generated by a machinelearning method such as deep learning, without extracting an edge. FIG.3 is a diagram schematically showing the contents of the processperformed by the eye detector 24. In the drawing, IM represents an imagein which the edge EG is superimposed on the captured image. In thisdrawing, the occupant sitting in the driver's seat DS is exclusivelyfocused on. The eye detector 24 first extracts a contour CT, by fittinga model such as an ellipse or an oval to an edge EG, as shown in theupper diagram of FIG. 3. Next, as shown in the middle diagram of FIG. 3,the eye detector 24 sets a nose detection window NM on the basis of thecontour CT, and detects a position of a nose bridge BN which is a partin which the edge is easily and clearly extracted in the nose detectionwindow NM. Next, as shown in a lower diagram of FIG. 3, the eye detector24 sets eye detection windows EWr and EW1 of a predetermined size on theright and left sides of the nose bridge BN in which the left and righteyes of the occupant should exist, on the basis of the position of thenose bridge BN, and detects at least a part of the eyes in the eyedetection window EW. As a result, the eye detection windows EWr and EW1are set at positions overlapping the left and right eyes of the occupantsitting in the driver's seat DS. Hereinafter, when the eye detectionwindows EWr and EW1 are not distinguished, in some cases, they may besimply referred to as an eye detection window EW. Although specificexamples of the process of “detecting at least a part of the eye” arevariously defined, in the following description, it is assumed that “atleast a part of the contour of the eyes is detected”. When detecting thecontour, the eye detector 24 detects the contour, for example, byfitting a curve model to the distribution of the edge EG.

The mouth detector 26 sets a mouth detection window MW (not shown) onthe basis of the position of the nose bridge BN, for example, using apart of the processing procedure of the eye detector 24, and detects atleast a part of the outline of the image of the mouth in the mouthdetection window MW. The contour of the mouth means, for example, alower end line of an upper lip and an upper end line of a lower lip.Instead of the aforementioned case, the mouth detector 26 may directlydetect a part of the contour of the mouth, by inputting a captured imageto a learned model generated by a machine learning method such as deeplearning.

The eye opening rate deriving unit 28 derives the eye opening rate ofthe eyes of the occupant, on the basis of a positional relationship of aplurality of feature points in the contour of the eye detected by theeye detector 24. The plurality of feature points include, for example, afirst feature point at an end portion of the contour of the eye on theside closer to the imager 10 in a lateral direction (corresponding toone of an outer canthus or an inner canthus), a second feature point atthe upper end portion, a third feature point at the lower end portion,and a fourth feature point at an end portion on the side away from theimager 10 in the horizontal direction (corresponding to the other of theouter canthus or the inner canthus). FIG. 4 is a diagram for explainingthe process of the eye opening rate deriving unit 28. In the drawing, P1is a first feature point, P2 is a second feature point, P3 is a thirdfeature point, and P4 is a fourth feature point.

The eye opening rate deriving unit 28 virtually moves, for example, avertical line L1 from the right end of the eye detection window EW tothe left inside the eye detection window EW, and sets an intersectionpoint at the time of first intersecting with the contour ECT of the eyesas the first feature point P1. The eye opening rate deriving unit 28virtually moves, for example, a horizontal line L2 downward from theupper end of the eye detection window EW inside the eye detection windowEW, and sets the intersection point at the time of first intersectingwith the contour ECT of the eyes as the second feature point P2. The eyeopening rate deriving unit 28, for example, virtually moves a horizontalline L3 upward from the lower end of the eye detection window EW insidethe eye detection window EW, and sets the intersection point at the timeof first intersecting with the contour ECT of the eyes as the thirdfeature point P3. The eye opening rate deriving unit 28, for example,virtually moves the vertical line L4 from the left end of the eyedetection window EW to the right in the eye detection window EW, andsets the intersection point at the time of first intersecting with thecontour ECT of the eyes as the fourth feature point P4.

Further, the eye opening rate deriving unit 28 derives the degree Ro ofeye opening of the occupant on the basis of the coordinates of the firstto fourth feature points P1 to P4. For example, the eye opening ratederiving unit 28 may set a ratio of an interval Dx between the verticallines L1 and L4 and an interval Dy between the horizontal lines L2 andL3 as a degree Ro of eye opening (see Formula (1)). The method ofobtaining the degree Ro of eye opening is not limited thereto, and anymethod may be used.Ro=Dx/Dy  (1)

Further, the eye opening rate deriving unit 28 derives the eye openingrate of the occupant on the basis of the degree Ro of eye opening. Theeye opening rate deriving unit 28 defines, for example, a degree Roiniof eye opening derived on the basis of a captured image from about afirst few minutes after the occupant gets into the vehicle as an eyeopening rate of 100 [%], and divides the degree Ro of eye openingderived thereafter by the reference degree Roini of eye opening toderive an eye opening rate α (see Formula (2)). The present invention isnot limited thereto, and when person authentication of the occupant isperformed, the reference degree of eye opening corresponding to 100% foreach occupant may be stored in the memory, and the reference degree ofeye opening for each occupant may be read from the memory and used forcalculation. A specific value may be set for the reference degree Roiniof eye opening, or a specific value may be used at first and may begradually adjusted to the average degree of eye opening of the occupant.α=MIN {Ro/Roini,100 [%]}  (2)

In the description so far, although the eye opening rate is derived onthe basis of the degree Ro of eye opening on an image plane, forexample, by preparing a three-dimensional model of the eye andperforming the above-described process after mapping two-dimensionallyfrom the eye model rotated depending on a face orientation angleestimated from the relationship between the contour CT and the nosebridge BN, the estimation accuracy of the eye opening rate can beimproved.

The condition estimator 30 digitizes (or encodes to indicate the stage)the drowsiness of occupant, for example, on the basis of the eye openingrate α derived by the eye opening rate deriving unit 28 and the movementof the contour of the mouth detected by the mouth detector 26, andoutputs the drowsiness of occupant to various in-vehicle devices 100.For example, the condition estimator 30 outputs a numerical valueindicating that the drowsiness of occupant increases as the eye openingrate α decreases, and outputs a numerical value indicating that thedrowsiness of occupant increases as the number of times a “yawn” isinferred from the movement of the mouth contour increases. The conditionestimator 30 digitizes the drowsiness of the occupant, for example, bycalculating a weighted sum of a reciprocal of the eye opening rate α andthe number of yawns in an observation period (see Formula (3)). InFormula (3), γ is an index value indicating the drowsiness of theoccupant, Cy is the number of yawns during the observation period, andβe and βm are coefficients indicating a ratio of reflecting each of thedetection result of the eye detector 24 and the result of the mouthdetector 26 in the estimation of the occupant's condition. Thecoefficients βe and βm are set in advance so that the sum becomes, forexample, 1. For the method of detecting a yawn, for example, since themouth greatly changes in the vertical direction when yawning, a methodmay be used in which a vertical length of the contour of the mouthdetected by the mouth detector 26 is compared with a threshold value,and when a condition in which the vertical length is equal to or largerthan the threshold value continues for a predetermined time or more, itis determined that a yawn has occurred.γ=βe·(1/α)+βm·Cy  (3)

Here, the condition estimator 30 changes the ratio between thecoefficient βe and the coefficient βm, on the basis of the detectionresult of the eye detector 24 or the result of a process (for example,the degree Ro of eye opening) performed on the basis of the detectionresult of the eye detector 24. Specifically, when a predetermined eventoccurs, the condition estimator 30 may change the coefficient βe and thecoefficient βm so that the coefficient βe decreases and the coefficientβm increases.

The following is a list of predetermined events in which the conditionestimator 30 becomes a trigger which changes the ratio between thecoefficient βe and the coefficient βm.

(1) The fitting rate of the edge EG to the contour of the eye is lowerthan a reference.

(2) The variation of the distribution of the edge EG to the fittingcurve is larger than a reference.

(3) The pupil cannot be distinguished or the white of the eye and theiris cannot be distinguished.

(4) The condition in which the degree Ro of eye opening is equal to orless than the predetermined degree has continued for a predeterminedtime or more.

FIGS. 5 to 7 are diagrams showing a part of a person in which apredetermined phenomenon is likely to occur. Because the person shown inFIG. 5 has long eyelashes and is prone to downcast eyes, the probabilityof falling under (1) to (3) is high. Since the person shown in FIG. 6has long eyelashes, the probability of falling under (1) or (2) is high.Since the person shown in FIG. 7 has very narrow eyes, the probabilityof falling under (3) or (4) is high. If the coefficients βe and βm,which are the specified values, are used for these persons, since thecondition of the eyes cannot be satisfactorily distinguished initially,there is a risk of deterioration of the accuracy of estimatingdrowsiness. For this reason, the condition estimator 30 changes theratio of reflecting each of the detection result of the eye detector 24and the result of the mouth detector 26 in the estimation of conditionof the occupant such that the coefficient βe decreases and thecoefficient βm increases in the case of handling the aforementionedphenomena. Therefore, the occupant observation device 1 can maintainhigh accuracy of the occupant condition estimation.

Here, although the condition estimator 30 may switch the coefficients intwo stages using a set of coefficients of specific values (initialvalues) and a set of coefficients in which the coefficient βm increasesas the number of corresponding events among the aforementioned (1) to(4) becomes larger, an amount of decrease in the coefficient βe may beincreased, or the amount of increase in the coefficient βm may beincreased.

FIG. 8 is a flowchart showing an example of the flow of a processperformed by the image processing device 20. First, the image processingdevice 20 acquires an image (captured image) captured by the imager 10(step S200).

Next, the eye detector 24 detects the occupant's eyes (step S204), andthe eye opening rate deriving unit 28 derives the eye opening rate (stepS206). At the same time, the mouth detector 26 detects the occupant'smouth (step S208).

Further, the condition estimator 30 determines whether theabove-described predetermined phenomena have occurred (step S210). Whenthe condition estimator 30 determines that a predetermined event has notoccurred, the condition estimator 30 performs condition estimation usingthe initial values of the coefficients βe and βm, and outputs theresults (step S212). On the other hand, when it is determined that apredetermined event has occurred, the condition estimator 30 performscondition estimation using βe changed to a smaller value and βm changedto a large value, and outputs the result (step S214).

According to the occupant observation device 1 of the embodimentdescribed above, since the condition estimator 30 changes the ratio ofreflecting each of the detection result of the eye detector 24 and thedetection result of the mouth detector 26 in the estimation of thecondition of the occupant, on the basis of the detection result of theeye detector 24 or the result of process performed on the basis of thedetection result of the eye detector, the accuracy of the conditionestimation of the occupant can be maintained at a high level.

According to the occupant observation device 1, when the condition inwhich the degree Ro of eye opening of the eyes of the occupant is equalto or less than the predetermined degree continues for a predeterminedtime or more, since the condition estimator 30 changes the ratio ofreflecting the detection result of the eye detector 24 and the detectionresult of the mouth detector 26 in the estimation of the occupant'scondition, the accuracy of the estimation condition of the occupant canbe maintained high even for an occupant with a narrow eye.

As described above, while the embodiments for carrying out the presentinvention have been described using the embodiments, the presentinvention is not limited to such embodiments at all, and variousmodifications and substitutions may be made without departing from thegist of the present invention.

What is claimed is:
 1. An occupant observation device comprising: animager configured to capture an image of a head of an occupant of avehicle; and a processor configured to: detect at least a part of eyesof the occupant in the image captured by the imager, based on adistribution of edges extracted from the image, an edge of thedistribution of edges being a pixel or a pixel group in which adifference in a pixel value between the pixel or the pixel group andsurrounding pixel values of surrounding pixels is greater than areference value; detect at least a part of a mouth of the occupant inthe image captured by the imager; and estimate a condition of theoccupant based on a result of the eye detection and a result of themouth detection, and output a result of estimation to an in-vehicledevice, wherein the processor changes a ratio of reflecting the resultof the eye detection and a ratio of reflecting the result of the mouthdetection based on the estimate of the condition of the occupant,wherein the processor reduces the ratio of reflecting the result of theeye detection and increases the ratio of reflecting the result to themouth detection in the estimation of the condition of the occupant,based on the result of the eye detection or a result of a processperformed based on the result of the eye detection.
 2. The occupantobservation device according to claim 1, wherein the processor detectsat least a part of a contour of the eyes of the occupant, derives an eyeopening rate of the eyes of the occupant, based on a positionalrelationship of a plurality of feature points in the detected contour,and estimates the condition of the occupant based on the eye openingrate of the eyes of the occupant and the result of the mouth detection,and the result of the process performed based on the result of the eyedetection is the eye opening rate of the eyes of the occupant.
 3. Theoccupant observation device according to claim 2, wherein when acondition in which the eye opening rate of the eyes of the occupant isequal to or less than a predetermined degree continues for apredetermined time or more, the processor changes the ratio ofreflecting the result of the eye detection and the ratio of the resultof the mouth detection in the estimation of the condition of theoccupant.
 4. The occupant observation device according to claim 2,wherein the condition of the occupant is a drowsiness of the occupant,and the processor digitizes the drowsiness of the occupant, bycalculating a weighted sum of a reciprocal of the eye opening rate andthe number of yawns in an observation period.
 5. The occupantobservation device according to claim 1, wherein the processor reducesthe ratio of reflecting the result of the eye detection, when a pupilcannot be distinguished or a white of the eye and an iris cannot bedistinguished.
 6. The occupant observation device according to claim 1,wherein the processor extracts edges using an edge extraction filter anddetects at least a part of the eyes of the occupant in the image on thebasis of the distribution of edges, and reduces the ratio of reflectingthe result of the eye detection, when a fitting rate of the edges to thecontour of the eye is lower than a reference, or a variation of thedistribution of the edges to the fitting curve is larger than areference.
 7. An occupant observation device comprising: an imagerconfigured to capture an image of a head of an occupant of a vehicle;and a processor configured to: detect, as a result of an eye detection,at least a part of eyes of the occupant in an image captured by theimager; detect, as a result of a mouth detection, at least a part of themouth of the occupant in the image captured by the imager; and estimatea condition of the occupant based the result of the eye detection andthe result of the mouth detection, wherein the processor changes a ratioof reflecting each of the result of the eye detection and the result ofthe mouth detection in an estimation of the condition of the occupant,based on the result of the eye detection or a result of a processperformed based on the result of the eye detection, and wherein theprocessor reduces the ratio of reflecting the result of the eyedetection, when a pupil cannot be distinguished or a white of the eyeand an iris cannot be distinguished.
 8. An occupant observation devicecomprising: an imager configured to capture an image of a head of anoccupant of a vehicle; and a processor configured to: detect, as aresult of a eye detection, at least a part of eyes of the occupant in animage captured by the imager; detect, as a result of a mouth detection,at least a part of the mouth of the occupant in the image captured bythe imager; and estimate a condition of the occupant based on the resultof the eye detection and the result of the mouth detection, wherein theprocessor changes a ratio of reflecting each of the result of the eyedetection and the result of the mouth detection in an estimation of thecondition of the occupant, based on the result of the eye detection or aresult of a process performed based on the detection result of the eyedetection; and wherein the processor extracts edges using an edgeextraction filter and detects at least a part of the eyes of theoccupant in the image based on a distribution of edges, and reduces theratio of reflecting the result of the eye detection, when a fitting rateof the edges to the contour of the eye is lower than a reference, or avariation of the distribution of the edges to the fitting curve islarger than the reference.